Skip to content

Feature - E-L solid particle models#1297

Closed
JA0307 wants to merge 2598 commits intoMFlowCode:masterfrom
JA0307:feature/moving-solid-particles
Closed

Feature - E-L solid particle models#1297
JA0307 wants to merge 2598 commits intoMFlowCode:masterfrom
JA0307:feature/moving-solid-particles

Conversation

@JA0307
Copy link

@JA0307 JA0307 commented Mar 9, 2026

Solid Particle Implementation for E-L Solver

This update expands upon the pre-existing (in development) E-L solver for bubble dynamics to include solid particle dynamics. This is in support of the PSAAP center, which requires the capability to model solid particle dynamics in MFC.

Type of change : New feature, Refactor

Testing

The solver has been tested by running various 2D/3D problems involving fluid-particle interactions, such as spherical blasts surrounded by a layer of particles, shock-particle curtains, collision tests, etc.

The inputs to the EL solid particle solver have all been turned on/off to verify they work independent of each other, and together.

The code has been tested for CPU and GPU usage. The GPU usage has been tested on Tuolumne.

Two new files have been added:

  1. m_particles_EL.fpp
  2. m_particles_EL_kernels.fpp

File 1 has the main particle dynamics subroutines. This initializes the particles, computes fluid forces, coupling terms, computes collision forces, enforces boundary conditions, and writes the data for post-processing.

File 2 has the gaussian kernel projection code and the subroutine to compute the force on the particle due to the fluid. This compute the quasi-steady drag force, pressure gradient force, added mass force, stokes drag, gravitational force. Models for the quasi-steady drag are implemented here.

Checklist

  • [IP] I added or updated tests for new behavior
  • [IP] I updated documentation if user-facing behavior changed

Details:
In progress

  • [N (IP)] GPU results match CPU results
  • [ Y (AMD)] Tested on NVIDIA GPU or AMD GPU

Details:
GPU results not matching CPU results. Still deciphering which part of the model this is due to.

AI code reviews

@coderabbitai full review

sbryngelson and others added 30 commits November 12, 2025 14:31
Co-authored-by: Daniel Vickers <danieljvickers@login11.frontier.olcf.ornl.gov>
Co-authored-by: Daniel Vickers <danieljvickers@login08.frontier.olcf.ornl.gov>
This PR is to expand the SG EoS to its general form.

Such a change is needed for the PC model to work on both 5 and 6 equation models
Including a suggestion from coderabbitai, effectively initializing qv_avg to input parameter on 's_compute_speed_of_sound'
sbryngelson and others added 27 commits February 26, 2026 21:12
Added configuration options to reduce walkthrough noise and remove 'Finishing Touches' section content.
…and MPI communication

Features:
- Lagrangian bubble movement with projection-based void fraction smearing
- Kahan summation for accurate void fraction boundary conditions
- Extended MPI communication for moving EL bubbles
- New 2D and 3D moving Lagrangian bubble examples
- Updated test cases and golden files
This commit adds normal collision forces. The point is to checkpoint the code before merging with upstream. Collisions are not yet ready for use. Need to fix beta buffer filling first.
Buffer communication of beta is consistent with how the bubble solver fills beta.

There is an initial implementation of collisions, but this is not yet ready for use.

Various files that were different from master were updated to match.
Total gaussian weight computation is now done in the particle dynamics loop

Removed WENO reconstruction of gaussian smeared particle quantities for gradient fields and using fornberg finite differences instead. WENO is overkill and a time sink.

Re-added fornberg finite difference weight computation.
Particle Solver:

1. Call rhs functions

1.1 Particle Dynamics subroutine
  GPU loop across all particles:
    - Compute fluid force on the particle
    - If two way coupled, spread particle force and volume fraction across surrounding cells with gaussian
    - If collisions are off, update particle velocities and acceleration

  - Finalize volume fraction field by filling buffers

  - If collisions are on, call collision subroutine to compute collision forces with nearby particles and with any solid boundaries. Then update particle velocities/acceleration
    - Collision subroutine:
    - Ghost particles temporarily added to arrays
    - GPU loop across all particles:
	- Check for overlap with neighbor cells, including ghost particles
        - Collision force for particle pair only computed once
        - Each particle gets checked for wall collision

    - Communicate forces applied to ghost particles

1.2 Particles Source Subroutine
  GPU loop across all domain cells:
    - Update fluid solver rhs with particle source terms for mass, momentum, energy

2. Call Particle RK update
   - Update particle positions/velocity
   - Transfer particles across ranks
   - Enforce boundary conditions on the particles. Some overlap is allowed with solid walls to allow the collision force to handle moving the particle away from the wall naturally.
   - Update particle volume fraction field (zero out the field variables, compute the gaussian contribution for normalization, apply the gaussian kernel, communicate beta field).

Optimization:

    - particle volume fraction and source term contributions at each cell are updated with an atomic update in the gaussian kernel function. Using particle cell linked lists makes this much slower.
    - Particle dynamics particle loop handles the fluid force and the gaussian kernel for projection all in one. This reduced the number of GPU kernels launched and reuses particle data already in memory.
@wilfonba
Copy link
Contributor

wilfonba commented Mar 9, 2026

The git history on this branch is trashed. @JA0307 you'll need to close this PR and run the following commands to extract just the relevant changes you've made and put them on a new branch, and then open a PR with that branch.

  # 1. Create a new branch from upstream/master                                                                                                                                                                                                                                         
  git checkout upstream/master -b MovingBubblesFresh-clean                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                        
  # 2. Apply the diff as a single commit (or a few logical commits)                                                                                                                                                                                                                     
  git diff upstream/master...feature:moving-solid-particles -- | git apply                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                        
  # 3. Stage, review, and commit                                                                                                                                                                                                                                                        
  git add -A                                                                                                                                                                                                                                                                            
  git commit -m "<description>"                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                        

I trashed the git history on the fork that you based this code on, which is why the commit count and line change count are so absurdly high. In the future, I would recommend not forking forks of MFC and instead working from the master branch. I know this was a bit of a special circumstance with the EL MPI code not being merged yet.

@JA0307
Copy link
Author

JA0307 commented Mar 9, 2026

Makes sense, thank you!

@JA0307 JA0307 closed this Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.